RoboFinch: A versatile audio‐visual synchronised robotic bird model for laboratory and field research on songbirds

Multimodal communication is ubiquitous in nature. Plants lure pollinators with colour and fragrance, animals combine sounds visual displays during courtship humans gesture while talking (Fröhlich & van Schaik, 2018; Halfwerk et al., 2019; Higham Hebets, 2013; Munoz Blumstein, 2020; Partan Marler, 1999; Ullrich 2016). These multimodal presumably evolved to increase signal salience, aid species recognition or improve cognitive processing of the intended receivers, which can all contribute attractiveness effectiveness (Halfwerk Hebets Papaj, 2005). The mechanisms underlying improved detection have been experimentally addressed past decade (Gomes 2016; 2014; Heuschele 2009; Ręk Magrath, 2017; Uetz 2011). However, whether why signals are thus easier recognise, learn memorise compared that only target a single sensory systems far from understood (Leavell 2018), part because these types experiments often require high standards stimulus control across multiple physical modalities for long periods time. Human speech bird song learned involving primarily auditory vocal motor (Bolhuis 2010; Doupe Kuhl, 1999). Additionally, production usually accompanied by concurrent cues (lip beak movements). importance integration human demonstrated ‘McGurk-effect’: when people exposed mismatch between audio video track an articulated syllable (e.g. hearing [ba] seeing face articulating [ga], most will hear [da]; McGurk Macdonald, 1976). Furthermore, infants as young 18 20 weeks already recognise correspondences specific unique orofacial movements associated them. This suggests aspects highly relevant learning, even without external reinforcement (Kuhl Meltzoff, 1982). Interestingly, many songbird juveniles sing better being able see their tutors. this due input social factors remains unknown too confounding variables typically experimental setups used tutoring (Eales, 1989; Nelson, 1997; White, 2001). To disambiguate relevance audio-visual learning setup required allows independent life-like manner. Robotic models allow components be controlled independently mimic animal behaviours realistic 3D settings (contra using e.g. screens; Romano 2019). also us test artificial combinations assess how receivers process different facilitating multisensory studies). lifelike helped demonstrate indeed influencing non-additive ways, not predicted reactions (Narins 2003; Patricelli Krakauer, Taylor Ryan, 2013). Setups study early and/or lifetime memory formation so developed applied, but potential uncover improves functions. Improved robotic approaches needed understand important functioning involved signalling. Here, we describe designed, constructed applied zebra finch model, named RoboFinch. We aimed create printed, colour-realistically painted model male whose head singing were deduced speed recordings males, realistically enough function ranging acquisition studies evolution. based RoboFinch on (Taeniopygia guttata) it behavioural, neurobiological genetic analyses learning. methods presented here easily transferred other species. Previous work finches uncovered parallels (Griffith 2021; Pfenning 2014). Traditionally, analysis stimuli has focused playbacks, sometimes supplemented components, such non-animated clay plasticine models, taxidermic mounts presentations Derégnaucourt Gobes Bolhuis, 2007; Varkevisser 2021). Only recently animated robots employed presence (Araguas 2022; Simon previous attempts however did synchronise sound movement development. focus earlier therefore emulate tutors (inanimate models) provide contingencies (operant activation) rather than focusing question multimodality. latter questions would more naturalistic representation singing, level account covariance related elements. Birds produce songs syrinx modulate them altering resonance properties tract (Beckers 2004; Elemans, Hoese 2000; Nowicki, 1987; Podos Riede Goller, Westneat 1993). Song modulation involves opening closing shape oropharyngal-oesophageal cavity. Producing covary posture provides clear cues. Beak gape example positively several frequency amplitude characteristics elements (Goller Ohms Williams, fast development requires recording live males at speeds levels synchronisation channels. measure speed, variation alignment movements, first high-speed camera (120 frames/s) film six adult undirected (not directed another bird). videos base motion files could drive our sync acoustic Next, scanned scan print bodies beaks RoboFinches hand-painted colours mixtures closely matched real plumage beaks. then tested responses presenting juvenile birds, either housed alone paired companion bird. After confirming typical interactive (see Section 3), tutees robot moving its out playback calls tutor. juveniles' exposure two treatments behaviour tutor, isolation, female juvenile. video-recorded staged bird-robot encounters analysed behavioural acceptance of, engagement companions Finally, subset qualitatively adulthood proof-of-principle tool dissect relative tutoring. was carried VU Amsterdam. measurements colony Institute Biology, Leiden University. Throughout, birds 13.5/10.5 h light/dark cycle, 20–22°C 45%–65% humidity. had ad libitum access commercial tropical seed mixture (Beyers, Belgium), cuttlebone, grit drinking water. three times week hardboiled eggs once germinated seeds, vegetables fruit. Experiments reviewed approved University Committee experimentation, Animal Welfare Body Centrale Commissie voor Dierproeven (CCD) Netherlands (permit number AVD1060020186606) cared accordance National (the Animals Act, Wod, 2014) European legislation (EU directive no. 2010/63/EU). handheld scanner (Eva, Artec3D). upper lower highest resolution prepared skull (ATOS 5X, gom; Figure S1A,B Appendix S1). scans combined adjusted computer aided design (CAD) program Catia V5R20 (Dassault Systèmes) separate body parts. included upper-and beak, left-and right side (split along longitudinal axis) front-and back axis). split neck throat, position steepest curvature (Figure S1C driving various parts separately. For mechanics coils commercially available toys 2019), called DigiBirds (Silverlit Toys Manufactory), allowed open close (up 5.0 mm, measured tip beak) rotate (from −30° +30°). designed mechanical connected CAD software Catia. coil placed inside attached via lever beak. fixed head, case finches. rotations body. moved left right. rods, one central axis front, body, resulting rotating mounted metal tube through ran wiring coils. pvc stand larger tube, platform. rotated Nema 17 stepping (Type: 17HS3404L23P1-X1; ACT Motor GmbH) outside plastic platform S5 Pololu Tic T825 USB Multi-Interface Stepper Controller. wooden perches adjacent approach inspect by. printed stereolithography printing (Form 2; Formlabs), uses laser cure solid isotropic liquid photopolymer resin (Grey Pro, Formlabs Resin). assembly checked filed off caused friction. fully assembled patterns light reflectance Supplementary Methods deduce recorded A cage (76 × 45 cm) afternoon day before recording. next morning, 08:00 11:00, until he sang full songs. Plexiglas window middle front cage. table attenuated room. cross perch always camera. Audio made Sennheiser MKH40 microphone hanging 50 cm above TASCAM DR-100MKiii recorder. sampling rate 96 kHz 16-bit resolution. Video Casio (EX-ZR3600, 120 fps, 12× optical zoom) door bell (70027 Heidemann), disturb triggered produced short, impulse-like clearly visible clapper touched bell, later preparations. At start each record fps up 12 min. filtered band-stop filter 0 425 Hz Praat (version 6.0.19). synchronised Vegas Pro 14.0). analyse tracking Tracker (open source physics, physlets.org). selected direction camera, allowing optimal pixels tracked tips S6A Head similar way, selecting forehead. subtracted y values normalised per (1 maximum, closed). x tracker displacement resembled furthest turn right). Data stored text file (Δt) 8.3 ms S6B S1 example). introductory notes followed five motifs (mean duration ± SD = 4.2 1.2 s, mean motive repetitions 3.9 0.8). sequences where contact just head. custom-built controller board, Ardunio (Adafruit 3405; Mouser electronics) Controller order movements. speaker (Blaupunkt, CB4500), amplifier (Devine MA-400), behind model. Playback operated custom-written LabVIEW (National Instruments) NUC PC (NUC7i3BNK; Intel Corporation). select soundfile obtained males. movement, filmed iPhone XR highspeed fps) small delays playback. As onset very distinct—song started note no minimal movement—we pauses syllables end motif alignment. temporal precision remained accurate throughout experiment S10 accurate, generation spatial became gradually less course experiment. may changes coil, linking 4). subtle change positioning lead element. intermediate size) influenced changing parameters detailed guideline GitHub, link below), moment constant readjustments, decided do logistical time constraints. purpose reported (pairing temporally vs. non-synchronous sound/moving treatment) drifting accuracy fortunately affect meaning nevertheless provided calling 1; Figures S8 S9 S1; comparison exemplar). 100 maximum ±5 mm heads +30° 180 degrees. Spectrophotoscopic readings cheeks, flanks colourations those S2 instruction guide containing code Labview well files, found Materials GitHub|Zenodo (https://github.com/SimonHubs/RoboFinch|https://doi.org/10.5281/zenodo.7520589). In pilot experiment, dyads finches, age 75 days post-hatching (dph) dyad consisting female. observed webcam sessions over initial scored frame differencing custom LabView Instruments; S3 second Subjects stayed home mother siblings 35 dph. they set-up, tutored four conditions sensitive period (between 65 dph): (1) (“AudioOnly”), (2) unrelated age-matched same tutee (“AudioFemale”), (3) positioned time-aligned (“RoboSync”), (4) after presentation session finished (“RoboAsync”). tutor tutees, treatment. treatments, cage, RoboFinch, corresponding set 74 dB (Fast, A, re μPa, Voltcraft SL-451) 30 cm. All received 6 daily lasted During session, played: songs, RoboAsync, created complete broadcasting half hour following session. broadcasted RoboSync treatment S7 design). blocks 10-min presentation. information setup, design, data statistics, S1. get impression reacted (aged dph), female, setup. mesh window. sit directly both groups spent robot, particular moving, When shortly cages, close-by what seemed higher incidence soon 1, S3). Following results, total phase (in post hatch) treatments. first—control—treatment individually (hereafter ‘AudioOnly’). second—control—treatment stimuli, together ‘AudioFemale’). females species, testing effect (that tutor) (Adret, 2004). third—experimental—treatment consisted providing ‘RoboSync’) fourth—experimental—treatment third except synchronised. Instead, ‘RoboAsync’). Male closest location, spend (LMMs; treatment: N 24 χ2 41.1, df 3, p < 0.001; 2a). near location speakers changed comparing 15 last depended (Interaction Period Treatment: 38.75, 0.0001; Tutees AudioFemale much (both 0.99; contrast, RoboAsync (z-value 7.52, 0.0001) AudioOnly group −2.61, 0.036), findings, zoomed further 27% 5% S4 broadcast immediately afterwards. was, however, mainly (interaction on/offset 35.61, 0.001, 3a). regularly practised development, significant (χ2 2.14, 0.54). practicing increased 11.64, 0.009; 2b). 3.52, 0.0017), whereas find statistically 0.23, 0.99). interaction onset- offset 10.76, 0.013; 3). input, about equal amount (AudioOnly, z-value 0.02, 1.0), (AudioFemale, −2.66, 0.03). groups, playback, synced (RoboSync group, −4.86, −2.30, 0.08), suggesting actively paid attention synchrony. There trend call lacked companion, preen (all > 0.16, 2), (no date, 0.13). Visual inspection sonograms dph showed overtly exposure, incorporating 4a–d). copy quality subject future analysis. Here report imitation Based high-quality validated (each gestures male) Our serve simultaneously present (and species) production. assessing interacted condition listened song, isolation company results show preferentially vicinity behave towards ways normally interact Strikingly, interrupted ongoing activities approached mirroring ‘active listening behavior’ (Liu revealed although need unimodal Behavioural observations behaviour, singing/motor practice, known beneficial (Pytte Suthers, rescue type groups. Males versus differences directions groups: there consistent singing. While females, shows must yet simply replacing companion. salience stimulus. observation sung differently non-synced (one two) line predictions arising tools share 2) help future. Various lines evidence support notion role integrated deserves furth